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Abstract 

The  distinguishing  feature  of  a  metasystem  is  middle¬ 
ware  that  facilitates  viewing  a  collection  of  large,  dis¬ 
tributed,  heterogeneous  resources  as  a  single  virtual  ma¬ 
chine,  where  each  user  of  the  metasystem  is  identified  by  a 
unique  metasystem-level  identity.  The  physical  resources 
of  the  metasystem  can  exist  in  multiple  administrative 
domains,  each  with  different  local  security  requirements 
and  authentication  mechanisms  (e.g.,  Kerberos,  public- 
key).  The  problem  this  paper  addresses  is  how  to  map 
the  metasystems-level  identity  to  an  appropriate  account 
on  each  local  physical  machine  for  the  purposes  of  pro¬ 
cess  creation,  such  that  the  access  control  and  authen¬ 
tication  policies  of  each  local  machine  are  not  violated. 
This  mapping  must  ensure  the  integrity  of  the  local  ma¬ 
chines,  must  ensure  the  integrity  of  the  metasystem  user’s 
data,  and  must  not  unnecessarily  burden  either  the  meta¬ 
system  users,  the  metasystem  system  administrator,  or  the 
local  machine  system  administrators.  Specific  examples 
are  drawn  from  experiences  gained  during  the  deployment 
of  the  Legion  metasystem.  For  example,  Legion  configu¬ 
rations  for  local  sites  with  different  access  control  mech¬ 
anisms  such  as  standard  UNIX  mechanisms  and  Kerberos 
are  compared.  Through  analysis  of  these  configurations, 
the  inherent  security  trade-offs  in  each  design  are  derived. 
These  results  have  practical  importance  to  current  and  fu¬ 
ture  metasystem  users  and  to  sites  considering  any  future 
inclusion  of  local  resources  in  a  global  virtual  computer. 
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1  Introduction 

The  emerging  widespread  introduction  and  use  of  giga¬ 
bit  wide  area  and  local  area  networks  have  the  potential  to 
transform  the  way  people  compute,  and  more  importantly 
the  way  they  interact  and  collaborate  with  one  another.  The 
increase  in  bandwidth  will  enable  the  construction  of  wide 
area  virtual  computers  called  metasystems  or  metacomput¬ 
ers  (see  Figure  1).  The  challenge  in  the  construction  of  the 
metasystem  layer  is  to  provide  the  illusion  of  a  single  vir¬ 
tual  machine  from  resources  that  may  consist  of  thousands 
of  heterogeneous  processors,  storage  systems,  databases, 
legacy  codes,  and  user  objects,  all  distributed  over  wide- 
area  networks  spanning  multiple  administrative  domains. 
The  physical  machines  that  a  site  chooses  to  participate  in 
a  metasystem  are  generally  still  available  for  use  by  the 
site’s  local  users,  so  it  is  important  that  the  metasystem’s 
impact  on  local  users  be  minimal. 

A  key  challenge  of  the  metasystem  is  to  provide  a  se¬ 
cure  environment  for  both  resource  owners  and  users.  Se¬ 
curity  services  provided  by  the  metacomputer  software  in¬ 
frastructure  are  not  unlike  traditional  services  required  in 
uniprocessor  operating  systems.  Users  must  authenticate 
themselves  to  the  “system”.  Users  must  not  be  allowed  to 
access  arbitrary  resources.  Logging  mechanisms  must  be 
used  to  hold  users  accountable  and  to  track  intruders. 

However,  a  solution  to  the  metacomputer  security  prob¬ 
lem  requires  significant  additions  over  the  uniprocessor  se¬ 
curity  solution  [5],  particularly  because  the  machines  that 
comprise  the  metacomputer  may  not  be  administered  by 
a  single  policy-making  entity.  A  single  metasystem  can 
span  multiple  organizations,  but  only  if  each  individual  or¬ 
ganization’s  requirements  with  respect  to  resources  owned 
by  that  organization  are  satisfied.  The  metasystem  must 
be  made  to  accommodate  local  policy,  over  which  the 
metasystem  designers  have  little  or  no  influence.  That  is, 
metasystem  developers  cannot  mandate  underlying  secu¬ 
rity  policy  or  mechanism;  the  challenge  is  to  adhere  to  and 
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Figure  1 .  General  Metasystem  Architecture 


support  local  security  policies  while  creating  an  easy-to- 
use  and  secure  environment  for  end  users. 

Thus,  in  a  metacomputing  environment,  the  security 
problem  can  be  divided  into  two  main  concerns: 

1.  Protecting  the  metacomputer’s  high-level  resources, 
services,  and  users  from  each  other  and  from  possi¬ 
bly  corrupted  underlying  resources  ( within  the  meta¬ 
system  layer  in  Figure  1) 

2.  Preserving  the  security  policies  of  the  underlying  re¬ 
sources  that  form  the  foundation  of  the  metacomputer 
and  minimizing  their  vulnerability  to  attacks  from  the 
metacomputer  level  (the  vertical  arrows  in  Figure  1) 

For  example,  restricting  who  is  able  to  configure  a 
metacomputer-wide  scheduling  service  would  fall  in  the 
first  category — its  solution  requires  metacomputer-specific 
definitions  of  identity,  authorization,  and  access  control. 
Meanwhile,  enforcing  a  policy  that  permits  only  those 
metacomputer  users  who  have  local  accounts  to  run  jobs 
on  a  given  host  falls  in  the  second  category,  and  it  might 
require  a  means  to  map  between  metacomputer  identities 
and  local  identities.  Satisfying  this  second  concern  is  vi¬ 
tally  important,  because  the  typical  physical  machine  that 
in  part  comprises  the  metacomputer  appears  identical  to 
other  local  machines  to  the  local  users  (its  participation 
in  the  metacomputer  should  be  completely  hidden  to  non¬ 
metacomputer  users). 

In  a  metasystem,  arbitrary  users  must  not  be  allowed  to 
allocate  resources  on  arbitrary  machines.  In  contrast,  al¬ 
lowing  this  is  analogous  to  a  local  sysadmin  granting  an  ac¬ 
count  to  any  person  without  an  evaluation  of  what  the  per¬ 
son  might  do  on  the  machine  if  an  account  were  granted! 
Sysadmins  rarely  allow  this  in  order  to  protect  the  integrity 
of  the  data  on  their  machines  and  to  ensure  that  resources 


will  be  available  to  legitimate  users.  Similarly,  local  sites 
will  choose  to  participate  in  a  metasystem,  but  contingent 
on  assurances  that  their  local  security  policies  will  not  be 
violated. 

The  specific  requirement  of  a  metasystem  that  this  pa¬ 
per  addresses  is  that  the  metasystem  must  only  create  pro¬ 
cesses  for  users  that  have  been  authenticated  and  are  au¬ 
thorized  to  use  the  particular  underlying  resources.  For 
example,  a  site’s  security  requirements  could  range  from 
not  restrictive  (allowing  any  metasystem  user  to  create  pro¬ 
cesses,  even  if  the  particular  metasystem  user  does  not 
have  an  account  on  the  computers  at  the  site)  to  very 
restrictive  (allowing  only  those  users  who  have  personal 
accounts  on  the  machines  on  their  site,  and  only  after 
Kerberos  authentication).  Note  that  in  all  cases,  only 
users  who  have  authenticated  themselves  to  the  metasys¬ 
tem  layer  are  allowed  to  create  processes.  A  challenge  in 
the  metasystem  is  to  efficiently  select  those  resources  that 
support  one  or  more  of  the  user’s  authentication  certificates 
or  credentials,  and  then  securely  present  the  appropriate 
certificates  or  credentials  with  minimal  (if  any)  direct  in¬ 
teraction  with  the  user. 

To  create  a  process  on  one  of  the  physical  machines  that 
comprise  the  metasystem,  the  unique  metasystem  iden¬ 
tity  must  be  mapped  to  an  appropriate  account  on  the  se¬ 
lected  physical  machine.  This  mapping  must  be  specific  to 
the  authentication  requirements  of  the  local  machine,  must 
only  be  performed  for  those  physical  machines  upon  which 
the  metasystem  user  is  allowed  to  create  processes  (a  lo¬ 
cal  system  may  only  be  accessible  by  a  subset  of  all  users 
of  the  metasystem),  and  must  be  securely  configured  and 
performed  (a  non-privileged  user  is  not  allowed  to  specify 
this  mapping).  In  addition,  it  is  our  belief  that  the  success 
of  a  metasystem  greatly  depends  on  its  ease  of  use  by  end- 
users,  so  the  impact  on  the  metasystem  user  must  be  kept  to 


a  minimum.  However,  there  must  be  a  balance  between  the 
user’s  desire  to  only  have  to  be  interactively  authenticated 
once  for  the  entire  metasystem  versus  a  system  adminis¬ 
trator  requiring  a  particular  form  of  strong  authentication 
that  is  unique  to  her  site.  In  other  words,  the  user  does 
not  want  to  have  to  manage  potentially  dozens  of  SecurlD 
cards,  smart  cards,  and  passwords,  while  the  system  ad¬ 
ministrator  of  a  government  supercomputer  certainly  will 
not  accept  a  one-size-fits-all  approach  for  authentication 
in  the  metasystem  that  places  the  authentication  require¬ 
ments  of  a  university  computer  on  the  same  level  as  her 
more  strategic  resources. 

The  contribution  of  this  paper  is  that  it  reveals  how  this 
metacomputing  problem  manifests  itself  and  is  solved  in 
the  context  of  Legion,  which  is  a  object-based  metasys¬ 
tem  project  discussed  in  Section  2.  It  presents  the  Legion 
solution  to  three  site-specific  requirements  to  authentica¬ 
tion  and  authorization  for  the  purposes  of  process  creation. 
In  the  first  scenario,  the  site  requires  standard  UNIX  ac¬ 
cess  control  (similar  to  most  UNIX  installations  at  univer¬ 
sities).  In  the  next  two  scenarios,  the  sites  require  Kerberos 
authentication  but  differ  in  their  specific  security  policies 
regarding  process  creation  after  Kerberos  authentication. 
The  mechanism  by  which  to  map  Legion  identity  to  local 
account  is  discussed  for  each  scenario.  The  “level  of  se¬ 
curity”  as  compared  to  ease  of  use  of  each  scenario  is  dis¬ 
cussed  in  terms  of  the  likelihood  and  ramifications  of  the 
compromise  of  user  credentials.  The  importance  of  this 
work  is  that  it  describes  the  solution  to  real-world  security 
problems  that  exist  as  a  result  of  users  who  increasingly 
create  programs  that  require  transparent,  secure  access  to 
multiple,  distributed,  heterogeneous  resources.  Sites  are 
eager  to  deploy  the  Legion  metasystem  software,  but  only 
after  receiving  assurances  that  their  local  security  policies 
are  satisfied.  Without  solutions  such  as  these  described  in 
this  paper.  Legion  cannot  be  deployed  on  a  large-scale  ba¬ 
sis. 

The  organization  of  this  paper  is  as  follows.  Section  2 
presents  the  Legion  mechanisms  for  security  that  are  in¬ 
dependent  of  any  underlying  host  system.  This  section 
covers  identity,  authentication,  authorization,  and  account¬ 
ability.  Section  3  describes  how  processes  are  started  and 
controlled,  and  how  files  are  written,  on  a  system  that  em¬ 
ploys  standard  UNIX  access  control.  Section  4  describes 
two  configurations  in  which  Legion  integrates  with  a  Ker¬ 
berized  host.  Section  5  describes  projects  that  are  related 
to  this  work.  Section  6  contains  the  conclusions. 

2  Security  provided  by  Legion  mechanisms 

Legion  [7,  8]  is  an  object-based  metasystem  with  a  goal 
of  supporting  millions  of  objects  spread  across  thousands 
of  machines.  Legion  executes  as  middleware — below  the 


end-user  applications  but  above  the  operating  system  (thus, 
the  Legion  infrastructure  itself  is  not  privileged  code).  Le¬ 
gion  provides  process  management,  inter-process  commu¬ 
nication,  persistent  storage,  a  single  unified  file  system, 
and  security  services.  Legion  supports  PVM,  MPI,  C,  For¬ 
tran,  a  parallel  C++,  Java  and  the  CORBA IDL.  Legion  can 
select  resources  for  use  by  applications  and  securely  co¬ 
ordinate  large-scale  application  execution,  eliminating  the 
need  for  the  end-user  to  explicitly  log  on  to  each  machine, 
FTP  files,  create  processes,  create  temporary  files,  etc.  A 
project  of  Legion's  size  and  scope  is  faced  with  numer¬ 
ous  technical  challenges,  all  related  to  managing  the  poten¬ 
tially  huge  number  of  underlying  heterogeneous  hardware 
and  software  platforms. 

In  the  Legion  system,  host  objects  represent  processing 
resources.  When  a  Legion  object  is  instantiated,  it  is  a  Host 
Object  that  actually  creates  a  process  to  contain  the  newly 
activated  object.  The  Host  Object  thus  controls  access  to 
its  processing  resource  and  can  enforce  local  policies,  e.g., 
ensuring  that  a  user  does  not  consume  more  processing 
time  than  allotted.  Vault  objects  in  Legion  represent  stable 
storage  available  within  the  system  for  containing  Object 
Persistent  Representations  (OPRs).  Just  as  Host  Objects 
are  the  managers  of  active  Legion  objects.  Vault  Objects 
are  the  managers  of  inert  Legion  objects.  For  example. 
Vaults  are  the  point  of  access  control  to  storage  resources, 
and  can  enforce  policies  such  as  file  system  allocations. 

The  security  model  for  Legion  differs  significantly  from 
that  of  conventional  systems.  A  Legion  “system”  is  re¬ 
ally  a  federation  of  resources  from  multiple  administrative 
domains,  each  with  its  own  separately  evaluated  and  en¬ 
forced  security  policies.  As  such,  there  is  no  central  kernel 
or  trusted  code  base  that  can  monitor  and  control  all  in¬ 
teractions  between  users  and  resources.  Nor  is  there  the 
concept  of  a  superuser — no  one  person  or  entity  controls 
all  of  the  resources  in  a  Legion  system. 

While  there  is  no  single  superuser,  in  practice  there  are 
a  few  privileged  users  who  control  access  to  the  core  re¬ 
sources  such  as  vaults  and  hosts.  These  privileged  users 
establish  trust  relationships  between  themselves,  and  use 
these  trust  relationships  upon  which  to  control  access  to 
their  respective  resources.  For  example,  assume  that  there 
are  two  sets  of  resources,  controlled  by  two  separate  privi¬ 
leged  users,  and  that  there  are  established,  non-overlapping 
sets  of  users  of  these  resources.  One  of  these  privileged 
users  may  agree  to  let  the  other’s  set  of  users  have  access 
to  her  resources,  as  long  as  a  reciprocal  relationship  is  also 
established.  It  is  a  subject  of  future  research  to  determine 
to  what  degree  this  privilege  can  be  dispersed  amongst  a 
larger  group  of  privileged  users  (with  decreasing  privilege) 
and  still  remain  an  operational  metasystem. 

The  purpose  of  this  section  is  to  provide  a  discussion 


of  Legion  security  mechanisms  that  is  independent  of  the 
underlying  security  mechanisms  and  policies  of  the  host 
systems.  The  material  in  this  section  is  presented  primarily 
to  establish  the  context  for  the  remaining  sections.  For 
more  details  regarding  these  mechanisms,  see  [5], 

2.1  Identity 

Identity  is  fundamental  to  higher-level  security  services 
such  as  access  control.  Every  Legion  object  is  identified  by 
a  unique,  multi-field,  location-independent  Legion  Object 
Identifier,  or  LOID.  One  of  the  LOID  fields  contains  se¬ 
curity  information  such  as  an  X.509  certificate  [3]  or  more 
simply  just  an  RSA  public  key.  The  X.509  certificate  in  the 
LOID  is  not  an  X9.57  attribute  certificate  [16],  but  rather 
an  ID  certificate  (also  referred  to  as  a  public-key  certifi¬ 
cate  [4]),  and  pairs  a  public  key  with  a  person’s  name,  or¬ 
ganization,  identification  of  the  public  key  algorithm,  and 
other  information.  A  certificate  may  be  signed  by  a  certi¬ 
fication  authority  (CA)  that  vouches  for  the  association  of 
the  key  with  the  identifying  information.  To  cover  the  case 
where  a  recipient  doesn't  recognize  the  CA,  the  CA’s  own 
certificate  can  be  chained  onto  the  certificate,  allowing  the 
CA’s  CA  to  be  the  basis  of  authority.  The  user’s  X.509 
ID  certificate  is  propagated  with  requests  and  method  calls 
made  directly  or  indirectly  on  behalf  of  the  user.  The  in¬ 
formation  in  the  certificate  is  used  when  making  entries  to 
access  logs.  While  it  is  sufficient  to  use  only  the  LOID, 
at  times  it  is  easier  to  simply  include  this  information  di¬ 
rectly  in  the  logs,  without  requiring  a  mapping  from  LOID 
to  user  identification  such  as  name,  organization,  etc. 

In  a  distributed  object  system  such  as  Legion,  the  user 
typically  accesses  resources  indirectly,  and  objects  need 
to  be  able  to  perform  actions  on  his  behalf.  One  way  in 
which  to  allow  intermediate  objects  to  request  services  on 
behalf  of  an  originating  object  is  to  give  the  intermediate 
objects  a  copy  of  the  private  key  of  the  originating  object, 
thus  providing  necessary  authentication  information.  This 
approach  is  clearly  insecure,  as  intermediate  objects  could 
then  maliciously  originate  operations  on  false  behalf  of  the 
originating  object.  An  alternative  approach  is  to  have  in¬ 
termediate  objects  call  back  to  the  user  or  his  trusted  proxy 
when  they  receive  access  requests  in  the  user’s  name.  This 
step  puts  control  back  in  the  user’s  hands.  There  are  sev¬ 
eral  drawbacks  to  this  approach,  though.  First,  the  fine- 
grain  control  afforded  by  authorization  callbacks  may  be 
mostly  illusory.  It  can  be  very  difficult  to  craft  policies 
for  a  user  proxy  (or  even  the  real  user  himself!)  that  are 
much  more  than  “grant  all  requests” — too  much  contextual 
and  semantic  information  is  generally  missing  from  the  re¬ 
quest.  Beyond  this  barrier,  callbacks  are  expensive  and  do 
not  scale  well.  In  Legion,  after  all,  every  object  represents 
a  resource  of  some  type,  and  a  callback  on  every  method 


call  would  be  a  crippling  performance  hit. 

The  intermediate  solution  between  these  approaches  is 
to  issue  credentials  (also  known  as  X509v3  certificates 
with  explicit  authorizations  or  X9.57  attribute  certificates) 
to  objects.  A  credential  is  a  list  of  rights  granted  by  the  cre¬ 
dential’s  maker,  presumably  the  user.  They  can  be  passed 
through  call  chains.  When  an  object  requests  a  resource,  it 
presents  the  credential  to  gain  access.  The  resource  checks 
the  rights  in  the  credential  and  who  the  maker  is,  and  uses 
that  information  in  deciding  to  grant  access.  There  are  two 
main  types  of  credentials  in  Legion:  delegated  credentials 
and  bearer  credentials.  A  delegated  credential  specifies 
exactly  who  is  granted  the  listed  rights,  whereas  simple 
possession  of  a  bearer  credential  grants  the  rights  listed 
within  it.  A  Legion  credential  specifies  the  period  the  cre¬ 
dential  is  valid,  who  is  allowed  to  use  the  credential,  and 
the  rights — which  methods  may  be  called  on  which  spe¬ 
cific  objects  or  class  of  objects.  The  credential  also  in¬ 
cludes  the  identity  of  its  maker,  who  digitally  signs  the 
complete  credential. 

2.2  Access  control 

Each  Legion  object  is  responsible  for  enforcing  its  own 
access  control  policy.  The  general  model  for  access  control 
is  that  access  is  only  available  through  method  calls,  and 
that  each  method  call  received  at  an  object  passes  through 
a  Mayl  layer  before  being  serviced  (see  Figure  2).  Mayl 
decides  whether  to  grant  access  according  to  whatever  pol¬ 
icy  it  implements.  If  access  is  denied,  the  object  will  re¬ 
spond  with  an  appropriate  security  exception,  which  the 
caller  can  handle  any  way  it  sees  fit. 

Mayl  can  be  implemented  in  multiple  ways.  The  triv¬ 
ial  Mayl  layer  just  allows  all  access.  Most  objects,  how¬ 
ever,  use  the  Legion-provided  default  Mayl  implementa¬ 
tion,  which  essentially  defines,  on  a  per-method  basis,  an 
allow  list  and  a  deny  list.  If  a  LOID  is  on  both  lists,  access 
is  denied.  The  entries  in  the  lists  are  the  LOIDs  of  callers 
that  are  granted  or  denied  the  right  to  call  the  particular 
method.  Default  allow  and  deny  lists  can  be  specified  to 
cover  methods  that  don't  have  their  own  entries. 

By  default,  an  object  is  created  with  an  ACL  that  is  a 
function  of  the  class  of  the  object  and  gives  read  access 
to  the  world  and  write  access  to  the  owner.  For  example, 
all  methods  that  involve  “reading”  are  allowed  to  be  ex¬ 
ecuted  by  users  that  have  authenticated  to  the  particular 
Legion  system.  The  owner  of  the  object  is  free  to  dynami¬ 
cally  modify  any  allow  list  or  any  deny  list  for  any  method 
-  including  the  method  that  is  executed  to  modify  these 
lists!  Usually,  these  lists  are  modified  through  the  use  of 
a  GUI.  Currently,  the  use  of  a  standardized  access  control 
language  such  as  GAA  [13]  is  being  investigated. 

When  a  method  call  is  received,  the  credentials  it  car- 


Figure  2.  Legion  Implementation  of  Access  Control 


ries  are  checked  by  Mayl  and  compared  against  the  access 
control  lists.  For  example,  in  the  case  of  a  delegated  cre¬ 
dential,  the  caller  must  have  included  proof  of  his  identity 
in  the  call  so  that  Mayl  can  confirm  that  the  credential  ap¬ 
plies.  Multiple  credentials  can  be  carried  in  a  call;  check¬ 
ing  continues  until  one  provides  access. 

2.3  Communication  between  Legion  objects 

A  method  call  from  one  Legion  object  to  another  can 
consist  of  multiple  Legion  messages.  Because  Legion  sup¬ 
ports  dataflow-based  method  invocation,  the  various  argu¬ 
ments  of  a  method  call  may  flow  into  the  target  as  mes¬ 
sages  from  several  different  objects.  A  message  from  one 
Legion  object  to  another  Legion  object  may  be  sent  with 
no  security,  in  private  mode,  or  in  protected  mode.  In 
both  private  and  protected  modes,  certain  key  elements  of 
a  message  (e.g.,  any  contained  credentials)  are  encrypted 
with  the  public  key  of  the  recipient  Legion  object.  In  pri¬ 
vate  mode  the  body  of  the  message  is  encrypted,  whereas 
in  protected  mode  only  a  Message  Authentication  Code 
(MAC)  is  generated  to  provide  an  integrity  guarantee.  To 
enable  faster  computations,  protected  mode  is  the  default, 
although  the  user  can  easily  change  this.  Unless  private 
mode  is  already  on,  protected  mode  is  selected  automati¬ 
cally  if  a  message  contains  credentials.  The  mode  selected 
for  use  by  an  originating  object  is  applied  for  all  messages 
indirectly  generated  as  a  result  of  the  originating  message. 
For  example,  a  user  can  select  private  mode  when  calling 
an  object.  The  calls  that  the  object  makes  on  behalf  of 
the  user  will  also  use  private  mode,  and  so  on  down  the 
line.  Currently,  encryption  is  based  on  the  RSA  toolkit 
(RSAREF  2.0). 

In  addition  to  protecting  credentials,  both  protected 
mode  and  private  mode  encrypt  a  computation  tag  con¬ 
tained  in  eveiy  Legion  message,  a  random  number  token 
that  is  generated  for  each  method  call.  All  the  messages 
that  make  up  a  given  method  call  contain  the  same  compu¬ 
tation  tag.  The  tag  is  used  to  assemble  incoming  messages 
from  multiple  objects  into  a  single  method  call  and  to  iden¬ 
tify  the  return  value  for  a  call  made  earlier.  If  an  attacker 
knows  the  computation  tag  for  a  method  call,  he  can  forge 


complete  messages  containing  arguments  or  return  values, 
even  without  holding  any  credentials.  The  computation  tag 
is  treated  as  a  shared  secret,  and  is  never  transmitted  in  the 
clear  unless  “no  security”  mode  is  selected. 

The  security  layer  does  not  provide  mutual  authentica¬ 
tion.  The  sender  can  be  assured  of  the  identity  of  the  re¬ 
cipient,  because  only  the  desired  recipient  can  read  the  en¬ 
crypted  parts  of  the  message.  The  recipient  usually  doesn’t 
care  who  the  actual  sender  is;  its  decisions  are  based  solely 
on  the  credentials  and  the  X.509  ID  certificate(s)  that  ar¬ 
rived  in  the  message. 

3  Legion  integration  with  standard  UNIX 
access  control 

The  previous  section  described  object  interactions  at  the 
logical  level  of  the  metasystem  in  Figure  1 — specifically, 
how  one  Legion  object  can  authenticate  with  another  Le¬ 
gion  object  and  exchange  secure  communication.  How¬ 
ever,  Legion  objects  must  physically  exist  on  a  host  that 
is  part  of  the  metasystem.  This  section  describes  how  the 
metasystem  identity  is  mapped  to  a  physical  account  and 
an  object  is  instantiated  on  a  host  that  requires  only  stan¬ 
dard  UNIX  access  control. 

Our  general  strategy  for  isolating  Legion  objects  from 
one  another  is  to  run  them  in  separate  accounts  on  the  host 
system.  The  accounts  that  can  be  used  for  this  purpose  fall 
into  two  categories: 

•  For  those  Legion  users  who  happen  to  have  accounts 
on  the  system,  objects  can  run  on  their  normal  user 
accounts. 

•  For  other  users,  there  is  a  pool  of  generic  accounts 
that  are  assigned  for  Legion  use. 

The  generic  accounts  usually  have  minimal  permissions. 
The  local  Host  Object  and  Vault  Object  also  have  their  own 
accounts. 

The  use  of  generic  accounts  is  appropriate  for  those  sites 
that  require  that  one  user  be  isolated  from  another  on  the 
underlying  host  machine  (e.g.,  different  UNIX  accounts) 


but  do  not  require  strict,  persistent  mapping  from  metasys¬ 
tem  ID  to  local  account.  In  these  situations,  accountability 
is  through  logging  Legion  user  X.509  ID  certificates.  The 
more  “tightly  controlled”  metasystems  will  probably  not 
make  use  of  generic  accounts,  if  only  because  policy  of¬ 
ten  mandates  that  no  person  can  make  use  of  a  resource 
without  first  filing  the  appropriate  account  applications. 

Object  creation  requests  arrive  at  the  Host  Object  as  nor¬ 
mal  method  invocations,  and  can  thus  be  controlled  using 
the  standard  Legion  access  control  mechanism  for  meth¬ 
ods.  For  each  request,  the  host  checks  the  credentials 
against  the  user  LOIDs  and  groups  that  are  allowed  to  cre¬ 
ate  objects  on  it.  If  everything  is  acceptable,  it  next  selects 
an  account  for  the  new  object  to  run  in;  depending  on  the 
credentials  in  the  creation  request  and  its  local  configu¬ 
ration,  it  may  choose  a  local  user  account  or  one  of  the 
generic  accounts.  The  accounts  are  subject  to  scheduling 
and  resource  control  just  like  CPU  time,  memory  usage, 
and  so  on;  an  object’s  lease  on  an  account,  especially  a 
generic  account,  is  usually  limited. 

All  Legion  objects  are  associated  with  some  persistent 
storage,  typically  in  the  form  of  a  directory  in  the  local  file 
system  managed  by  the  Object  Vault.  Before  starting  an 
actual  process  for  the  new  object  in  the  allocated  account, 
the  host  needs  to  change  the  ownership  of  the  object’s  di¬ 
rectory  from  the  vault  user-id  to  the  newly  allocated  user- 
id.  The  location  of  the  directory  that  will  contain  the  new 
object’s  persistent  state  is  passed  to  the  host  as  part  of  the 
activation  request  (this  location  was  obtained  through  a 
method  on  the  local  vault  performed  by  the  object’s  cre¬ 
ator,  likely  its  class).  Ownership  of  this  directory  must  be 
changed  both  to  protect  the  object’s  state  from  access  by 
other  objects  (which  will  run  under  different  user-ids),  and 
to  make  the  state  accessible  to  the  new  object. 

Finally,  the  host  needs  to  spawn  the  actual  process  that 
will  execute  the  object  on  the  appropriate  account.  To 
carry  out  this  step,  and  to  change  ownership  of  the  object’s 
persistent  state,  the  host  requires  access  to  some  privileged 
operations.  However,  the  host  does  not  execute  with  root 
permissions.  Access  to  these  required  privileged  opera¬ 
tions  is  encapsulated  in  a  process  control  daemon  (PCD) 
that  executes  on  the  host,  providing  services  to  the  Host 
Object  in  a  controlled  fashion.  The  PCD  is  a  small,  easily 
vetted  program  that  runs  with  root  permissions.  It  is  con¬ 
figured  only  to  allow  access  by  the  host  account.  Two  of 
its  key  functions  are  to  permit  changing  directory  owner¬ 
ship  and  to  create  new  processes  on  a  designated  account. 
The  PCD  limits  the  accounts  for  which  this  can  be  done  to 
a  set  configured  by  the  local  system  administrator.  The  set 
includes  the  generic  Legion  accounts  and  potentially  the 
accounts  of  local  Legion  users. 

A  simplified  example  of  this  operation  is  shown  in  Fig¬ 


ure  3.  In  this  example.  Object  12,  which  is  executing  on 
Fred’s  account  on  some  machine  in  the  metasystem,  and 
has  access  to  Fred’s  Legion  credentials,  wants  to  create  an 
object  on  the  host  Elmer.virginia.edu.  To  do  this.  Object  12 
asks  the  Host  Object  executing  on  Elmer  to  start  a  process 
for  the  new  object.  First,  the  Host  Object  confirms  that  Ob¬ 
ject  12  is  acting  on  Fred’s  behalf  by  looking  at  the  creden¬ 
tials  contained  in  the  request.  Then,  the  Host  Object  maps 
a  request  by  Fred  to  a  generic  account  {Legion- generic- 1) 
on  Elmer.virginia.edu  that  has  been  established  at  the  time 
that  Legion  was  installed  on  Elmer.virginia.edu .  Finally, 
the  Host  Object  asks  the  PCD  to  spawn  a  new  process  as 
Legion-generic-1  for  the  new  object. 

As  the  PCD  starts  the  object  running,  the  host  logs  an  au¬ 
dit  trail  using  the  X.509  ID  certificate  for  the  user  whose 
credentials  accompanied  the  request.  The  audit  trail  pro¬ 
vides  essential  information  if  the  new  object  misuses  local 
resources.  If  the  object  has  exceeded  its  use  of  local  re¬ 
sources,  the  host  can  request  that  the  PCD  kill  it  directly. 
When  an  object  loses  or  relinquishes  its  use  of  an  account, 
the  Host  Object  uses  the  PCD  to  change  the  ownership  of 
its  persistent  state  back  to  the  Vault  Object.  If  the  object 
is  reactivated  later  on  a  different  account,  ownership  of  the 
state  can  be  changed  to  the  appropriate  local  user-id.  After 
an  account  is  reclaimed,  the  PCD  terminates  all  processes 
running  on  it  and  generally  cleans  it  up. 

Security  Analysis.  In  accepting  this  approach,  a  sysad¬ 
min  at  a  local  site  is  trusting  the  Legion  software  to  le¬ 
gitimately  map  Legion  identities  to  local  accounts  (if  the 
PCD  is  configured  to  map  to  non-generic  local  accounts). 
If  the  Legion  credentials  for  a  particular  user  are  stolen, 
the  risk  to  the  system  is  less  when  configured  for  generic 
accounts  than  with  non-generic  accounts  (by  their  nature, 
when  a  user  is  finished  with  a  generic  account,  no  persis¬ 
tent  state  remains).  A  negative  aspect  of  this  approach  is 
that  a  site  must  install  the  PCD  as  privileged  code,  cre¬ 
ating  a  potential  point  of  attack  for  intruders.  However, 
this  code  has  been  vetted  by  numerous  experts,  increas¬ 
ing  the  confidence  on  the  part  of  local  site  regarding  the 
safety  of  this  code.  Of  course,  if  the  Legion  Host  Object 
account  (“Legion- Account”  in  Figure  3)  is  cracked,  the  in¬ 
truder  can  create  processes  under  the  accounts  of  any  local 
Legion  users. 

Overall.  The  PCD-based  implementation  is  sufficient 
for  many  local  system  administrators.  Legion  authenti¬ 
cation  is  used  to  determine  who  gains  access  to  local  re¬ 
sources,  and  the  resources  made  available  are  also  con¬ 
strained  to  those  usable  from  a  limited  set  of  accounts. 
Detailed  logging  provides  accountability.  The  safety  of 
credentials  is  a  chief  design  goal  in  the  security  architec- 


Object  12 
"Fred" 


1)  CreateObject(-) 


2)  Authenticate 


Change  ownership  to 
"Legion-generic- 1 " 


Host  Object:  Elmer 
"Legion-Account" 


(3)  Map  "Fred"  to 
^  "Legion-generic- 1" 


PCD:  Elmer 


Spawn  process  as 
^  "Legion-generic- 1" 


Elmer.virginia.edu 


Figure  3.  Object  Creation  on  a  Standard  UNIX  Access  Control  Host 


ture  and  mechanisms  of  Legion.  An  alternative,  simple 
approach  is  to  have  all  Legion  objects  execute  under  the 
“Legion-Account”  account.  In  general,  we  have  found 
that  sysadmins  do  not  like  this  approach  because  of  lim¬ 
ited  accountability — as  far  as  they  see,  only  one  account 
“does  anything”  with  regard  to  Legion.  We  (the  Legion  de¬ 
signers)  do  not  advocate  this  approach,  because  it  does  not 
provide  the  necessary  isolation,  as  all  files  and  processes 
are  owned  by  one  oner  id  (meaning  that  one  Legion  user 
can  use  UNIX  mechanisms  to  destroy  or  subvert  another 
Legion  user  on  the  same  host). 

4  Legion  integration  with  a  Kerberized  host 

Increased  security  concerns  have  caused  many  sites  to 
switch  from  standard  UNIX  access  control  to  the  use  of 
Kerberos  [11],  Kerberos  is  a  trusted  third-party  authenti¬ 
cation,  in  which  users  and  services  register  their  keys.  In 
this  paper,  familiarity  with  the  basic  Kerberos  protocols 
are  assumed.  Legion  has  been  integrated  with  a  Kerber¬ 
ized  host,  and  is  discussed  in  this  paper,  because  certain 
sites  have  wanted  to  run  Legion  and  mandated  the  use  of 
Kerberos. 

It  is  important  to  understand  that  the  “Kerberized  Host” 
in  this  section  refers  to  a  host  that  is  executing  the  MIT 
source  code  distribution  [9].  This  paper  does  not  discuss 
efforts  to  integrate  Kerberos  directly  with  public  key  cryp¬ 
tography  [14],  because  this  paper  focuses  on  integration 
with  widely-deployed  Kerberos  systems.  Similarly,  the  use 
of  Proxiable  tickets  in  Kerberos  is  not  discussed,  because 
their  usage  is  not  widely  supported.  While  its  support  is 
not  directly  discussed,  much  of  the  discussion  is  applica¬ 
ble  to  AFS. 

In  this  section,  assume  that  a  simple  Legion  metacom¬ 


puter  is  being  constructed.  There  are  only  two  machines 
involved:  Khost,  which  is  the  Kerberized  Host  machine, 
and  NKhost,  which  is  a  machine  that  does  not  require  Ker¬ 
beros  authentication  (instead,  it  uses  only  password-based 
authentication).  Additionally,  it  is  useful  to  define  the  fol¬ 
lowing  entities: 

L-creds  credentials  that  are  necessary  to  function  in  the 
Legion  virtual  computer;  these  credentials  reflect  a 
metasystem  user’s  identity 

K-creds  Kerberos  credentials;  obtained  via  Kerberos 
kinit  either  explicitly  or  implicitly 

L-admin  a  particular  metasystem  user  with  some  admin¬ 
istrative  duties  for  the  metasystem;  Note  that  the 
“L-admin”  identity  means  nothing  directly  on  either 
Khost  or  NKhost  (it  must  be  mapped  onto  an  ac¬ 
count  on  each  machine);  by  definition,  L-admin  has 
L-creds;  in  order  to  authenticate  to  the  metasystem 
layer,  the  L-admin  user  must  have  first  authenticated 
to  the  underlying  physical  machine:  if  L-admin  first 
logged  into  Khost,  then  the  user  associated  with  L- 
admin  has  K-creds  as  well. 

Legion-Khost  an  account  on  Khost  that  is  used  solely 
for  executing  processes  related  to  the  metasystem; 
“Legion-Khost”  is  not  an  identity  recognized  at  the 
metasystem  level  (“Legion-Khost”  has  K-creds  asso¬ 
ciated  with  it  but  does  not  automatically  have  L-creds 
associated  with  it) 

Alice-KL  a  person  who  has  an  account  on  Khost  and 
wishes  to  participate  in  Legion  virtual  machine  ex¬ 
ecuting  on  Khost;  has  K-creds;  has  L-creds;  has  an 
account  on  NKhost 


Bob-L  a  person  who  has  an  “account”  on  the  Legion 
virtual  machine  but  no  account  on  Khost;  has  L- 
creds;  does  not  have  K-creds;  has  a  UNIX  account 
on  NKhost 

4.1  Kerberos  background:  .k51ogin  and  .k5users 

In  Kerberos,  there  is  the  capability  to  allow  one  principal 
to  grant  access  to  another  principal  (after  the  other  princi¬ 
pal  has  authenticated).  The  file  .k51ogin  allows  one  user 
to  unconditionally  grant  another  user  the  ability  to  spawn 
processes  as  the  first  user.  For  example,  if 'Bob/.lologin 
contained  ”Jim”,  user  Jim  could  “ksu  Bob”  and  have  a 
running  shell  whereas  new  processes  are  tagged  as  being 
owned  by  Bob.  Note  that  in  this  case,  Jim  does  not  ac¬ 
quire  Bob’s  credentials;  rather,  in  this  case,  Jim,  executing 
as  UID  Bob,  has  a  copy  of  the  Kerberos  credentials  Jim 
had  immediately  prior  to  executing  ksu.  .k5users  is  more 
restrictive  than  .k51ogin;  .k5users  lets  Bob  allow  Jim  to 
execute  only  certain  binaries  as  Bob,  for  instance  “/bin/ls”. 
The  entry  in'Bob/.lousers  in  this  case  is  “Jim  /bin/ls”.  In 
this  case,  Jim  cannot  execute  a  shell  as  Bob;  Jim  could 
only  “ksu  Bob  -e  /bin/ls”. 

4.2  Kerberos  solution  #1:  k51ogin 

A  simple  solution  to  allow  a  user  such  as  Alice-KL  with 
an  account  on  Khost  to  access  the  metasystem  is  to  add 
“Legion-Khost”  to  her  .k51ogin  file.  This  approach  allows 
Legion-Khost  to  execute  “binary  1”  as  Alice-KL  by  invok¬ 
ing  “ksu  Alice-KL  -e  full-path-to-binaryl”.  For  Alice-KL 
to  start  a  process  on  Khost,  essentially  the  same  steps  as 
in  Figure  3  are  taken.  The  difference  in  this  situation  is 
that  the  PCD  does  not  exist,  as  the  Host  Object  can  cre¬ 
ate  processes  directly  as  Alice-KL  via  invocations  of  ksu. 
In  this  configuration.  Legion  mechanisms  will  ensure  that 
Bob-L  will  not  be  able  to  start  new  processes  on  Khost. 
However,  Bob-L  will  be  able  to  use  services  (processes) 
of  Alice-KL,  but  only  if  Alice-KL  has  configured  Legion 
authorization  mechanisms  to  let  Bob-L. 

Security  Analysis.  The  analysis  consists  of  a  number  of 
cases: 

If  the  Legion-Khost  account  is  compromised  (i.e.,  an 
attacker  obtains  Legion-Khost  Kerberos  credential 
cache,  or  breaks  into  the  Host  Object  and,  for 
example,  causes  the  Host  Object  to  execute  a  binary 
that  allows  the  credential  cache  to  be  read),  then 
all  Legion  users  on  Khost,  even  if  they  have  never 
used  Legion,  have  been  compromised.  The  attacker 
cannot  get  their  K-creds,  but  can  start  processes  on 
their  accounts1.  There  is  a  variation  of  this  approach, 

1  The  attacker  can  get  their  K-creds  if  the  user  is  logged  in. 


beyond  the  scope  of  this  paper,  that  uses  .k5users 
instead  of  .k51ogin  in  order  to  reduce  the  scope  of 
attack  if  the  Legion-Khost  is  compromised.  However, 
this  approach  is  substantially  more  complicated  to 
implement  and  analyze. 

If  Alice-KL’s  K-creds  are  stolen  (i.e.,  either  by  some  ac¬ 
tivity  irrespective  of  Legion,  or  if  Alice-KL  gives  her 
K-creds  to  a  Legion  “con-artist”  object),  then  only 
Alice-KL's  account  is  compromised.  In  this  case,  an 
attacker  can  replace  legitimate  Alice-KL  service  with 
corrupted  service,  thus  tricking  Bob-L  if  Bob-L  wants 
to  use  the  service.  However,  the  scope  of  this  attack 
may  include  several  machines,  as  Kerberos  creden¬ 
tials  are  typically  valid  at  a  potentially  large  set  of 
machines. 

If  Alice-KL’s  L-creds  are  stolen  Alice-KL’s  account  is 
compromised  because  the  Host  Object  can  be  asked 
to  start  jobs  on  her  account. 

If  L-admin’s  L-creds  are  stolen  an  attacker  can  effec¬ 
tively  shut  down  the  Host  Object  but  not  break  se¬ 
curity.  Note:  admin  has  no  special  privileges  with  re¬ 
gard  to  the  Host  Object,  beyond  being  able  to  change 
the  ACLs  on  the  Host  Object. 

Overall.  The  fact  that  Alice-KL  can  start  a  process  on 
the  Khost  without  directly  obtaining  and  presenting  K- 
creds  is  both  positive  and  negative:  The  use  of  the  Legion 
virtual  machine  by  Alice-KL  is  easier  and  perhaps  more 
secure  because  she  does  not  need  to  directly  acquire  Ker¬ 
beros  credentials.  A  potential  problem  is  that  the  Legion- 
Khost  has  unlimited  access  to  the  Khost  account  of  Alice- 
KL.  For  this  reason,  Alice-KL  and/or  the  sysadmins  of 
Khost  might  require  that  Alice-KL  get  a  separate  account 
on  Khost  for  use  with  the  Legion  virtual  machine.  For 
many  installations,  this  approach  is  sufficient,  satisfying 
the  requirement  of  Kerberos  authentication  before  use  of 
the  physical  resources  (the  authentication  in  this  case  is  by 
the  Legion-Khost  principal).  Security  is  provided  by  the 
Legion  mechanisms  based  on  L-creds  that  have  timeouts, 
recovery  mechanisms,  and  potentially  very  specific  scope 
and  privilege.  It  is  also  very  easy  to  implement. 

4.3  Kerberos  solution  #2:  KProxy  object 

In  general,  a  problem  of  the  k51ogin  approach  is  that 
a  user  must  grant  unlimited  access  to  her  account  by  the 
Legion-Khost  principal.  A  second  problem  is  that  a  user 
such  as  Alice-KL  (or  more  precisely  a  person  imperson¬ 
ating  Alice-KL)  does  not  have  to  authenticate  to  the  KDC 
of  the  Khost  Kerberos  realm  in  order  to  use  the  physical 


resources.  A  second  approach  eliminates  these  problems, 
but  at  the  cost  of  simplicity. 

The  essential  component  of  the  design  is  a  Legion 
KProxy  Object  for  each  user.  This  KProxy  Object  securely 
holds  the  Legion  user’s  Kerberos  credentials.  The  KProxy 
Object  for  user  Fred  executes  under  Fred’s  UID  on  a  ma¬ 
chine  upon  which  Fred  has  an  account.  Whenever  a  Flost 
Object  anywhere  in  the  metasystem  wants  to  create  an  ob¬ 
ject  on  Fred’s  behalf  on  its  associated  physical  machine, 
the  Flost  Object  performs  a  call  back  to  the  KProxy  Ob¬ 
ject  for  Fred  to  obtain  a  valid  ticket  for  that  particular  host. 
Fred’s  KProxy  Object  will  only  issue  Fred’s  Kerberos  cre¬ 
dentials  if  Fred's  valid  Legion  credentials  are  presented 
in  the  request  (more  generally,  the  access  control  mech¬ 
anisms  of  Fred's  KProxy  Object  can  be  configured  to  is¬ 
sue  Fred’s  Kerberos  credentials  to  any  object  that  presents 
valid  credentials  on  behalf  of  any  user  with  whom  Fred  has 
previous  established  a  trust  relationship).  The  Host  Object 
creates  the  new  object  via  a  call  to  ksu,  without  requiring 
the  use  of  the  .k51ogin  file. 

Note  that  all  communication  of  secret  information  is 
either  done  via  Kerberos  mechanisms  (DES)  or  Legion 
mechanisms  (RSAREF).  Cross-realm  authentication  is  im¬ 
mediately  and  transparently  supported  in  this  design:  kinit 
only  has  to  be  performed  once  for  each  group  of  Kerberos 
realms  that  support  cross-realm  authentication  with  each 
other.  The  Legion  KProxy  object  will  automatically  ob¬ 
tain  Ticket  Granting  Tickets  (TGTs)  for  the  other  realms 
based  on  the  existence  of  a  valid  TGT  for  a  given  host. 

The  steps  that  Alice-KL  must  take  in  order  to  create 
her  KProxy  object  are  shown  in  Figure  4.  First,  Alice- 
KL  obtain  her  K-creds  from  machine  NKhost  (this  can 
also  be  performed  on  Khost,  although  it  doesn't  have  to 
be).  By  default,  these  K-creds  are  tied  to  the  IP  ad¬ 
dress  of  NKhost.  On  NKhost,  Alice-KL  then  executes  le- 
gion_create_kproxy,  which  asks  the  Host  Object  on  Khost 
to  create  an  instance  of  KProxy -class  on  Khost.  As  part  of 
this  (not  shown),  Alice-KL  interacts  with  the  KDC  to  ob¬ 
tain  a  ticket  that  is  usable  from  Khost.  These  K-creds  are 
then  used  by  the  Host  Object  in  an  invocation  of  ksu  to  ac¬ 
tually  create  the  KProxy  object  that  will  hold  Alice-KL’s 
K-creds.  This  KProxy  object  will  execute  on  this  machine 
under  Alice-KL’s  account.  Note  that  neither  Alice-KL’s 
.k51ogin  nor  .k5users  contains  an  entry  that  directly  al¬ 
lows  Legion-Khost  to  execute  this  ksu;  instead,  ksu  is  in¬ 
voked  with  an  explicit  copy  of  Alice-KL’s  K-creds.  Now, 
anytime  in  the  future  that  the  Host  Object  on  Khost  wants 
to  create  an  object  on  Alice-KL’s  behalf,  it  interacts  with 
this  KProxy  object  to  obtain  a  valid  ticket  for  use  in  the 
ksu  invocation.  Additionally,  any  Host  Object  in  the  realm 
will  interact  with  Alice-KL’s  KProxy  object  when  creating 
objects. 


Why  can’t  the  Legion  user  obtain  K-creds  for  a  par¬ 
ticular  computation  before  starting  the  computation,  thus 
eliminating  the  need  for  the  KProxy  object?  Legion  is 
an  object-based  system  in  which  the  necessary  function¬ 
ality  to  start  and  coordinate  large-scale  computation  may 
be  spread  across  hundred  or  thousands  of  objects.  When 
a  user  originates  computation,  the  user  has  no  idea  on 
what  machines  processes  ultimately  will  be  started  either 
directly  or  indirectly  on  behalf  of  this  user  request.  For  ex¬ 
ample,  the  user  may  attempt  to  “run  discreet-simulation- 1 
1000  times”.  At  this  point,  the  user  has  no  idea  (he  may 
not  care)  which  machines  are  then  selected  by  scheduler 
objects.  Thus,  when  the  user  originates  computation,  he 
cannot  obtain  tickets  (which  are  tied  to  IP  addresses)  for 
all  of  the  machines  on  which  processes  will  be  started.  Our 
approach  uses  a  call  back  to  the  KProxy  object  to  get  the 
appropriate  ticket  for  each  machine. 

A  second,  related  note  is  that  the  user  may  not  be 
present  (i.e,  logged  onto  a  machine  in  the  Kerberos  realm) 
when  the  Legion  software  attempts  to  spawn  processes  on 
his  behalf.  For  example,  assume  that  the  user  starts  a 
long-running  activity  on  one  machine,  and  then  logs  off. 
The  Legion  software  might  then  need  to  spawn  a  process 
on  another  machine  (if,  for  example,  the  computation  is 
pipelined,  where  each  element  of  the  pipeline  is  a  suffi¬ 
ciently  long  computation).  Again,  we  need  a  call  back 
to  something  (the  KProxy  object  in  our  approach)  in  or¬ 
der  to  obtain  the  necessary  ticket.  One  approach  might 
be  to  somehow  get  a  ticket  from  the  process  that  already 
exists  that  is  performing  the  active  part  of  the  pipelined 
computation  (because  the  credentials  cache  is  present  for 
this  process,  and  presumably  it  contains  a  TGT).  This  is 
essentially  the  approach  of  the  KProxy  object. 

The  key  limitation  that  this  approach  overcomes  is  that 
a  process  can  only  be  started  on  Khost  as  user  Alice-KL  if 
Alice-KL  has  previously  authenticated  herself  to  the  KDC 
of  the  Kerberos  realm.  In  addition,  at  any  point,  Alice-KL 
can  control  the  creation  of  processes  under  her  account  by 
either  limiting  the  lifetime  of  the  ticket  held  by  her  KProxy 
Object,  or  eliminate  her  KProxy  object  completely.  How¬ 
ever,  it  is  still  the  case  that  the  Legion  user  must  trust  the 
Legion  software  to  create  processes  only  that  she  intended. 

Security  Analysis.  The  analysis  consists  of  a  number  of 
cases: 

If  the  Legion-Khost  account  is  compromised  If  Alice- 
KL  uses  the  subverted  Khost,  the  Host  Object  can  use 
her  L-creds  to  obtain  the  K-creds  from  the  KProxy. 
The  K-creds  can  then  be  misused  by  the  Host  Ob¬ 
ject,  for  example,  to  spawn  arbitrary  processes  under 
Alice-KL’s  user  id  (such  as  “rm  *”).  Note  that  un¬ 
like  the  k51ogin  approach,  where  a  subverted  Khost 
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Figure  4.  Object  Creation  on  Kerberos-Controlled  Host 


Legion  account  can  immediately  abuse  all  local  Le¬ 
gion  user  accounts,  this  approach  limits  the  attacker 
to  misusing  the  accounts  of  users  currently  starting 
objects  on  the  Khost.  This  allows  intrusion  detection 
as  an  approach  for  limiting  the  damage  caused  by  an 
attacker. 

If  Alice-KL’s  K-creds  are  stolen  If  the  K-creds  are 
stolen  in  some  activity  irrespective  of  the  metasys¬ 
tem,  the  same  ramifications  exist  as  the  k51ogin 
approach.  If  Alice-KL’s  K-creds  are  stolen  by 
attacking  her  KProxy  object,  the  duration  of  the 
compromise  is  limited  by  the  direct  or  indirect  scope 
of  the  K-creds  held  by  the  KProxy  object,  which  can 
be  controlled  by  Alice-KL. 

If  Alice-KL’s  L-creds  are  stolen  The  same  ramifications 
as  the  k51ogin  approach. 

If  admin’s  L-creds  are  stolen  The  same  ramifications  as 
the  k51ogin  approach. 

Overall.  This  approach  trades  off  some  additional  com¬ 
plexity  in  terms  of  the  systems  structure  and  some  extra 
effort  on  the  part  of  users  (who  periodically  need  to  upload 
refreshed  Kerberos  credentials  to  their  respective  KProxy 
object)  for  an  added  measure  of  attack  containment.  This 
approach  meets  the  requirement  that  the  user  actively  au¬ 
thenticate  through  the  Kerberos  mechanism  before  using 
the  local  resources  (unlike  Kerberos  k51ogin  approach). 
This  approach  incurs  added  overhead  due  to  the  call-backs 
to  the  KProxy  object;  however,  these  call-backs  only  occur 
at  the  time  of  object  creation,  so  the  impact  should  not  be 
significant.  Note  that,  as  opposed  to  the  krlogin  approach, 
the  duration  and  scope  of  an  attacker’s  compromise  is  in 
general  limited  by  the  K-creds  being  held  by  the  KProxy 
objects,  which  can  be  easily  controlled. 


5  Related  work 

There  are  several  projects  being  conducted  to  support 
inter-operability  between  a  particular  security  infrastruc¬ 
ture  and  Kerberos,  for  example  supporting  a  single  login 
for  NetWare  and  Kerberos  [1],  This  project  has  similar 
goals  to  Legion’s  integration  with  Kerberos — in  particu¬ 
lar,  no  changes  to  Kerberos  and  no  reduction  in  security  in 
either  security  realm  due  to  the  single  login.  A  significant 
difference  between  the  Legion  project  and  these  projects  is 
that  Legion  attempts  to  build  security  mechanisms  the  can 
be  viewed  as  being  on  top  of  underlying  security  mech¬ 
anisms  of  host  systems,  whereas  these  projects  generally 
attempt  to  support  single  sign-on  of  co-existing  realms.  In 
a  metasystem,  it  cannot  be  expected  that  every  realm  or 
administrative  domain  directly  acknowledge  and  support 
each  other’s  existence. 

Minsky  and  Ungureanu  address  the  need  of  unifying  het¬ 
erogeneous  security  policies  in  distributed  systems  by  in¬ 
troducing  a  formalism  that  describes  various  security  poli¬ 
cies  [10].  A  unified  mechanism  is  used  to  enforce  the  secu¬ 
rity  schemes.  This  work  is  important  for  the  construction 
and  analysis  of  security  policies  in  metasystems,  in  that 
the  metasystem  mechanism  must  support  a  wide  variety  of 
local  policies.  However,  in  metasystems,  the  approach  is 
that  the  local  sites  can  have  whatever  security  policy  they 
want,  and  it  is  very  likely  that  it  will  not  be  specified  for¬ 
mally.  Requiring  every  local  site  to  specify  their  security 
policies  in  a  single  formalism  is  difficult  if  not  impossible, 
severely  impeding  a  metasystem’s  deployment. 

Yialelis  and  Sloman  describe  a  security  framework  for 
object-based  distributed  systems  [17].  This  project  is  re¬ 
lated  to  the  work  in  Legion,  because  it  attempts  to  allow  the 
development  of  secure  distributed  applications  on  operat¬ 
ing  systems  with  varying  degrees  of  security  mechanisms 
built  in.  While  this  work  is  similar  to  the  Legion  mech¬ 
anisms  described  in  Section  2,  the  work  of  Yialelis  and 
Sloman  is  CORBA-based  and  does  not  address  the  general 


metasystems  requirements,  such  as  hardware  heterogene¬ 
ity  and  multiple  administrative  domains. 

Globus  [6]  is  another  metasystem  research  project,  and 
as  such  is  addressing  many  of  the  same  issues  as  Legion. 
In  many  instances,  convergent  evolution  has  led  to  simi¬ 
lar  solutions  to  these  problems.  For  example.  Globus  has 
a  small,  easily-verified  module  called  the  Gatekeeper  that 
runs  as  root  and  is  responsible  for  remote  process  manage¬ 
ment,  in  much  the  same  manner  as  the  PCD.  The  manner 
in  which  Globus  integrates  with  Kerberos  is  through  use  of 
the  Generic  Security  Services  API  (GSS-API  [15]).  The 
level  of  granularity  of  the  GSS-API  and  the  Legion  ob¬ 
ject  model  are  fundamentally  different:  In  GSS-API,  two 
applications  such  as  FTP  and  FTPD  establish  a  security 
context  and  then  communicate  based  on  the  security  con¬ 
text.  In  Legion,  objects  are  significantly  more  fine-grained 
than  applications  such  as  FTP  and  FTPD — the  overhead  to 
establish  contexts  establishing  and  deleting  security  con¬ 
texts  for  each  pair  of  communicating  objects  is  intuitively 
too  expensive,  as  the  number  of  object-object  communica¬ 
tions  is  potentially  quite  large  in  the  life  of  a  computation. 

CRISIS  [2]  is  the  security  architecture  for  the  WebOS 
project  at  UC  Berkeley.  The  WebOS  provides  many  of  the 
same  high-level  services  as  Legion.  WebOS  is  fundamen¬ 
tally  different  than  Legion  in  that  while  WebOS  focuses 
on  system-level  support  for  building  and  running  wide- 
area  applications.  Legion’s  goals  are  to  provide  an  object- 
based  programming  model  suitable  for  such  a  wide-area 
application.  The  principle  goals  of  CRISIS  are  similar  to 
the  goals  of  the  security  architecture  in  Legion:  to  use  re¬ 
dundancy  to  reduce  the  likelihood  of  system  compromise, 
cache  whenever  possible  to  improve  performance,  support 
fine-grained  control  over  delegated  rights,  make  extensive 
use  of  logging,  support  local  autonomy,  and  to  make  the 
design  as  simple  as  possible.  A  difference  between  the  two 
systems  is  a  result  of  Legion’s  support  for  autonomy  which 
is  not  a  focus  of  WebOS:  Legion  supports  dynamically- 
configured  local  security  mechanisms,  and  CRISIS  sup¬ 
ports  uniform  mechanism  across  all  of  the  nodes  of  the 
wide-area  system  (although  policy  within  each  node  may 
be  separately  defined). 

6  Conclusions 

While  a  goal  of  a  metasystem  can  be  to  provide  an 
abstraction  of  a  single  virtual  machine  from  large,  dis¬ 
tributed,  heterogeneous  resources,  ultimately  a  metasys¬ 
tem  computation  is  comprised  of  activities  on  physical  ma¬ 
chines  at  different  sites  with  different  goals.  These  activ¬ 
ities  include  process  creation,  file  creation  and  modifica¬ 
tion,  “native”  IPC  mechanisms,  “native”  schedulers  such 
as  Codine  and  LoadLeveler,  etc.  This  paper  described  the 
issues  involved  in  supporting  the  notion  of  identity  at  the 


level  of  the  metasystem,  specifically  with  regard  to  pro¬ 
cess  creation.  It  was  shown  how  a  metasystem  identity  can 
be  securely  mapped  to  a  physical  account  both  for  stan¬ 
dard  UNIX  access  control  and  for  Kerberos-based  access 
control.  In  the  future,  other  authentication  systems  will  be 
integrated  with  current  Legion  authentication  support.  A 
particular  system  that  is  currently  being  developed  is  inte¬ 
gration  with  SSL/TLS  via  OpenSSL  [12]. 

On  a  practical  level,  these  approaches  are  important  be¬ 
cause  they  are  in  active  use  at  various  local  sites  within  de¬ 
ployed  Legion  networks.  Many  times,  a  single  concurrent 
computation  can  consist  of  many  processes  spread  across 
hosts  that  require  different  authentication  and  access  con¬ 
trol.  Therefore,  the  security  implications  of  each  are  of 
great  interest  to  system  administrators  and  users  at  such 
sites.  On  a  more  general  level,  these  designs  demonstrate 
the  degree  to  which  the  Legion  architecture  can  accommo¬ 
date  and  adapt  to  site-local  requirements. 

The  Legion  system  is  currently  widely  deployed,  in¬ 
corporating  diverse  resources  at  Supercomputing  Cen¬ 
ters,  Labs,  and  Universities.  For  more  information 
about  the  current  status  of  the  Legion  system,  see 
http://legion.virginia.edu.  The  power  of  the  environment 
increases  with  the  scale  and  scope  of  the  system.  We  con¬ 
tinue  to  actively  integrate  new  sites  into  Legion.  A  nat¬ 
ural  part  of  the  evolution  requires  us  to  adapt  the  Legion 
security  architecture  to  new  site-local  policies  and  mech¬ 
anisms.  The  work  presented  here  describes  our  current 
dominant  site  configurations.  In  the  future  we  expect  to 
see  this  set  expand  as  Legion  deployment  increases. 
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